Methods for determining if teleoperation should be disengaged based on the user&#39;s gaze

ABSTRACT

A method for disengaging a surgical instrument of a surgical robotic system comprising receiving a gaze input from an eye tracker; determining, by one or more processors, whether the gaze input indicates the gaze of the user is outside or inside of the display; in response to determining the gaze input indicates the gaze of the user is outside of the display, determining an amount of time the gaze of the user is outside of the display; in response to determining the gaze of the user is outside of the display for less than a maximum amount of time, pause the surgical robotic system from a teleoperation mode; and in response to determining the gaze of the user is outside of the display for more than the maximum amount of time, disengage the surgical robotic system from the teleoperation mode.

BACKGROUND Field

Embodiments related surgical robotic systems, are disclosed. Moreparticularly, embodiments related to surgical robotic systems andcorresponding methods for determining if a teleoperation mode of thesurgical robotic system should be disengaged based on a user's gaze, aredisclosed.

Background

Endoscopic surgery involves looking into a patient's body and performingsurgery inside the body using endoscopes and other surgical tools. Forexample, laparoscopic surgery can use a laparoscope to access and viewan abdominal cavity. Endoscopic surgery can be performed using manualtools and/or a surgical robotic system having robotically-assistedtools.

A surgical robotic system may be remotely operated by a surgeon tocommand a robotically-assisted tool located at an operating table. Suchoperation of a robotically-assisted tool remotely by a surgeon may becommonly referred to as teleoperation or a teleoperation mode. Forexample, the surgeon may use a computer console located in the operatingroom, or it may be located in a different city, to command a robot tomanipulate the surgical tool mounted on the operating table. Therobotically-controlled surgical tool can be an endoscope mounted on asurgical robotic arm. Accordingly, the surgical robotic system may beused by the remote surgeon to perform an endoscopic surgery.

The surgeon may provide input commands to the surgical robotic system,and one or more processors of the surgical robotic system can controlsystem components in response to the input commands. For example, thesurgeon may hold in her hand a user input device such as a joystick or acomputer mouse that she manipulates to generate control signals to causemotion of the surgical robotic system components, e.g., an actuator, asurgical robotic arm, and/or a surgical tool of the robotic system.

SUMMARY

For a teleoperated surgical robotic system with an open display in whichthe user can view their surroundings (as compared to a periscope typedisplay) there is the possibility that the surgeon is looking away fromthe screen but still holding the user input devices (UlDs) that controlthe robotic tools. This introduces a risk since the surgeon could movethe UlDs and unintentionally move the tools while not focusing on thescreen. The instant disclosure describes methods for determining whetherthe teleoperation mode should be paused or disengaged based on a gaze ofthe user. For example, the system includes an eye tracker attached to athree dimensional (3D) display which can detect if the surgeon islooking off screen and can pause teleoperation as soon as the surgeon isdetermined to be looking away. If additional conditions are met (such asmoving the UlDs while looking away or looking away for longer than apredetermined time) teleoperation may be disengaged. If teleoperation isdisengaged the system may be configured to require that the surgeonactively engage again after looking back at the screen. If teleoperationis only paused, the system may be configured to allow the surgeon toimmediately take control of the tools once they look back at the screen,without any further action.

In some aspects, the method may include what are referred to herein as agaze-off-screen operation and a gaze-teleop-interlock operation. Thegaze-off-screen operation may determine if a user is looking off screenor not. The gaze-teleop-interlock may determine when teleoperationshould be paused and when teleoperation should be disengaged in part,based on the outcome of the gaze-off-screen operation. For example, inthe gaze-off-screen operation, the system determines if a user islooking at the screen or not based on gaze data collected by an eyetracker. Representatively, based on the data collected by the eyetracker, the gaze-off-screen operation determines the probability thatthe user is looking at the screen (and therefore teleoperation mode cancontinue or be engaged) or that the user is looking off screen (andtherefore teleoperation mode should be paused or disengaged). Exemplarydata and/or information that may be evaluated in the gaze-off-screenoperation may include the following:

(1) f both eyes are invalid (e.g, not detected by the tracker) or ifboth are valid (e.g., detected by the tracker) but the distance betweentheir calculated screen points is too large, the probability that theuser may not be looking at the screen increases;

(2) If at least one eye is valid and inside (e.g., within the bounds ofthe display), this indicates that the tracker is certain that at leastone eye is looking inside. The probability that the user is not lookingat the screen decreases and/or is low, or said another way, theprobability of the user looking at the screen is determined to be high;

(3) If both eyes are valid and outside or one valid and outside, thisindicates that the tracker is certain that no eye is looking inside andat least one is looking outside. The probability of the user not lookingat the screen increases and/or is high, or said another way, theprobability that the user is looking at the screen decreases and/or islow.

In another embodiment, an exemplary gaze-off-screen operation mayinclude the following:

(1) Determine if the gaze point is a fixation or a saccade (e.g., rapidmovement between fixation points). This is done for the left and rightgaze point;

(2) If at least one eye is fixating and inside, the probability of beingon-screen increases.

(3) If at least one eye is fixating and outside, the probability ofbeing off-screen increases.

(4) If both eyes are in a saccade, the probability of being off-screenincreases.

Exemplary gaze-teleop-interlock scenarios for determining whether, basedon the gaze-off-screen analysis, the teleoperation mode should bedisengaged, paused and/or engaged are as follows: (1) If the user looksaway for more than a blink or quick glance, pause teleoperation, andwhen the user looks back at the screen, s/he can immediately control thetools; (2) If the user looks away for a significant amount of time, orthe user moves the teleoperation controllers (UID) a significant amountwhile looking away, disengage teleoperation mode. Once disengaged, theuser will need to actively engage again to take control of the tools.

Representatively, in one aspect, the invention is directed to a methodfor disengaging a surgical instrument of a surgical robotic system. Themethod may include receiving a gaze input from an eye tracker thattracks a gaze of a user relative to a display associated with thesurgical robotic system and determining, by one or more processorscommunicatively coupled to the eye tracker, whether the gaze inputindicates the gaze of the user is outside or inside of the display. Inresponse to determining the gaze input indicates the gaze of the user isoutside of the display, the method determines an amount of time the gazeof the user is outside of the display. In response to determining thegaze of the user is outside of the display for less than a maximumamount of time, the method may pause the surgical robotic system from ateleoperation mode such that a user interface device of the surgicalrobotic system is prevented from controlling the surgical instrumentuntil a gaze input indicating the gaze of the user is inside of thedisplay is received. In addition, in response to determining the gaze ofthe user is outside of the display for more than the maximum amount oftime, the method may include disengaging the surgical robotic systemfrom the teleoperation mode such that the user interface device isprevented from controlling the surgical instrument until an activeengagement input is received. In addition, determining the gaze inputindicates the gaze of the user is outside of the display may includereceiving a valid gaze input indicating that a gaze for both eyes of theuser are detected by the eye tracker; and determining a distance betweenthe gaze for both eyes is outside of a maximum distance associated witha size of the display. Still further, the aspect of determining the gazeinput indicates the gaze of the user is outside of the display mayinclude receiving an invalid gaze input indicating a gaze of at leastone eye of the user is undetectable by the eye tracker; and determininga gaze path or a gaze location of the at least one eye is moving towarda border of the display. Moreover, the aspect of determining the gazeinput indicates the gaze of the user is outside of the display mayinclude receiving a head location input from a head tracker associatedwith the surgical robotic system; and determining, based on the headlocation input, a nose of the user is not facing the display or a faceof the user is undetectable. In addition, determining the gaze of theuser is inside the display may include receiving a valid gaze inputindicating that a gaze of at least one eye of the user is detected bythe eye tracker; and determining a location of the gaze is within thedisplay. The method may further include receiving a movement input fromthe user interface device of the surgical robotic system; and inresponse to receiving the movement input when the surgical roboticsystem is paused, disengage the surgical robotic system. In someaspects, the active engagement input indicates a user is facing thedisplay, a surgical chair is in a particular orientation, or the userinterface devices is in a particular orientation. In some aspects, themaximum amount of time may be more than 100 milliseconds.

In another aspect, a surgical robotic system. The surgical roboticsystem may include a surgical instrument, a user console comprising adisplay, an eye tracker for tracking a gaze of a user with respect tothe display, and a user interface device, and one or more processorscommunicatively coupled to the surgical instrument and the user console.The processor may ber configured to: receive a gaze input from the eyetracker; determine whether the gaze of the user is outside or inside ofthe display based on the gaze input; and pause and/or disengage thesurgical robotic system from a teleoperation mode when the user gaze isoutside the display such that the user interface device is preventedfrom controlling the surgical instrument. The display may be an opendisplay including a screen and a frame surrounding the screen. The userinterface device may be a portable handheld user input device that ismechanically ungrounded with respect to the user console. The system mayfurther include a head tracker, and determining whether the gaze of theuser is outside or inside the display is based on a head location inputfrom the head tracker. In addition, the gaze of the user may bedetermined to be outside of the display when a gaze for both eyes of theuser are detected by the eye tracker; and a distance between the gazefor both eyes is outside of a maximum distance associated with a size ofthe display. In some aspects, the gaze of the user is determined to beoutside of the display when a gaze of at least one eye of the user isundetectable by the eye tracker; and a gaze path or a gaze location ofthe at least one eye is moving toward a border of the display. In otheraspects, the gaze of the user is determined to be outside of the displaywhen a head location input from a head tracker associated with thesurgical robotic system indicates a nose of the user is not facing thedisplay or a face of the user is undetectable. Still further, the gazeof the user is determined to be inside the display when a gaze of atleast one eye of the user is detected by the eye tracker; and a locationof the gaze is within the display. In some aspects, the system furtherincludes a user interface device motion sensor, and the surgical roboticsystem is paused and/or disengaged from a teleoperation mode when amotion of the user interface device greater than a maximum translationis detected. In some aspects, in response to determining the gaze of theuser is outside of the display for less than a maximum amount of time,the surgical robotic system is paused from a teleoperation mode suchthat a user interface device of the surgical robotic system is preventedfrom controlling the surgical instrument until a gaze input indicatingthe gaze of the user is inside of the display is received. In someaspects, the maximum amount of time may be more than 100 milliseconds.In response to determining the gaze of the user is outside of thedisplay for more than a maximum amount of time, the surgical roboticsystem may be disengaged from the teleoperation mode such that the userinterface device is prevented from controlling the surgical instrumentuntil an active engagement input is received.

The above summary does not include an exhaustive list of all aspects ofthe present invention. It is contemplated that the invention includesall systems and methods that can be practiced from all suitablecombinations of the various aspects summarized above, as well as thosedisclosed in the Detailed Description below and particularly pointed outin the claims filed with the application. Such combinations haveparticular advantages not specifically recited in the above summary.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example andnot by way of limitation in the figures of the accompanying drawings inwhich like references indicate similar elements. It should be noted thatreferences to “an” or “one” embodiment of the invention in thisdisclosure are not necessarily to the same embodiment, and they mean atleast one. Also, in the interest of conciseness and reducing the totalnumber of figures, a given figure may be used to illustrate the featuresof more than one embodiment of the invention, and not all elements inthe figure may be required for a given embodiment.

FIG. 1 is a pictorial view of an example surgical robotic system in anoperating arena, in accordance with an embodiment.

FIG. 2 is a pictorial view of a display and gaze tracker, in accordancewith an embodiment.

FIG. 3 is a block diagram of an exemplary operation for disengaging ateleoperation mode based on a user gaze, in accordance with anembodiment.

FIG. 4 is a block diagram of an exemplary operation for determiningwhether a user gaze is outside a display, in accordance with anembodiment.

FIG. 5 is a pictorial view of an exemplary operation for determiningwhether a user gaze is outside a display, in accordance with anembodiment.

FIG. 6 is a pictorial view of an exemplary operation for determiningwhether a user gaze is outside a display, in accordance with anembodiment.

FIG. 7 is a block diagram of a computer portion of a surgical roboticsystem, in accordance with an embodiment.

DETAILED DESCRIPTION

In various embodiments, description is made with reference to thefigures. However, certain embodiments may be practiced without one ormore of these specific details, or in combination with other knownmethods and configurations. In the following description, numerousspecific details are set forth, such as specific configurations,dimensions, and processes, in order to provide a thorough understandingof the embodiments. In other instances, well-known processes andmanufacturing techniques have not been described in particular detail inorder to not unnecessarily obscure the description. Reference throughoutthis specification to “one embodiment,” “an embodiment,” or the like,means that a particular feature, structure, configuration, orcharacteristic described is included in at least one embodiment. Thus,the appearance of the phrase “one embodiment,” “an embodiment,” or thelike, in various places throughout this specification are notnecessarily referring to the same embodiment. Furthermore, theparticular features, structures, configurations, or characteristics maybe combined in any suitable manner in one or more embodiments.

In addition, the terminology used herein is for the purpose ofdescribing particular aspects only and is not intended to be limiting ofthe invention. Spatially relative terms, such as “beneath”, “below”,“lower”, “above”, “upper”, and the like may be used herein for ease ofdescription to describe one element's or feature's relationship toanother element(s) or feature(s) as illustrated in the figures. It willbe understood that the spatially relative terms are intended toencompass different orientations of the device in use or operation inaddition to the orientation depicted in the figures. For example, if thedevice in the figures is turned over, elements described as “below” or“beneath” other elements or features would then be oriented “above” theother elements or features. Thus, the exemplary term “below” canencompass both an orientation of above and below. The device may beotherwise oriented (e.g., rotated 90 degrees or at other orientations)and the spatially relative descriptors used herein interpretedaccordingly.

As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising” specify the presence of stated features, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, steps, operations,elements, components, and/or groups thereof.

The terms “or” and “and/or” as used herein are to be interpreted asinclusive or meaning any one or any combination. Therefore, “A, B or C”or “A, B and/or C” mean “any of the following: A; B; C; A and B; A andC; B and C; A, B and C.” An exception to this definition will occur onlywhen a combination of elements, functions, steps or acts are in some wayinherently mutually exclusive.

Moreover, the use of relative terms throughout the description maydenote a relative position or direction. For example, “distal” mayindicate a first direction away from a reference point, e.g., away froma user. Similarly, “proximal” may indicate a location in a seconddirection opposite to the first direction, e.g., toward the user. Suchterms are provided to establish relative frames of reference, however,and are not intended to limit the use or orientation of any particularsurgical robotic component to a specific configuration described in thevarious embodiments below.

Referring to FIG. 1, this is a pictorial view of an example surgicalrobotic system 100 in an operating arena. The surgical robotic system100 includes a user console 102, a control tower 103, and one or moresurgical robots 120, including robotic arms 104 at a surgical roboticplatform 105, e.g., an operating table, a bed, etc. The system 100 canincorporate any number of devices, tools, or accessories used to performsurgery on a patient 106. For example, the system 100 may include one ormore surgical tools 107 used to perform surgery. A surgical tool 107 maybe an end effector that is attached to a distal end of a surgical arm104, for executing a surgical procedure.

Each surgical tool 107 may be manipulated manually, robotically, orboth, during the surgery. For example, the surgical tool 107 may be atool used to enter, view, or manipulate an internal anatomy of thepatient 106. In an embodiment, the surgical tool 107 is a grasper thatcan grasp tissue of the patient. The surgical tool 107 may be controlledmanually, by a bedside operator 108; or it may be controlledrobotically, via actuated movement of the surgical robotic arm 104 towhich it is attached. The robotic arms 104 are shown as a table-mountedsystem, but in other configurations the arms 104 may be mounted in acart, ceiling or sidewall, or in another suitable structural support.

Generally, a remote operator 109, such as a surgeon or other operator,may use the user console 102 to remotely manipulate the arms 104 and/orthe attached surgical tools 107, e.g., teleoperation or teleoperationmode. The user console 102 may be located in the same operating room asthe rest of the system 100, as shown in FIG. 1. In other environmentshowever, the user console 102 may be located in an adjacent or nearbyroom, or it may be at a remote location, e.g., in a different building,city, or country. The user console 102 may comprise a seat 110,foot-operated controls 113, one or more handheld user input devices, UID114, and at least one user display 115 that is configured to display,for example, a view of the surgical site inside the patient 106. In theexample user console 102, the remote operator 109 is sitting in the seat110 and viewing the user display 115 while manipulating a foot-operatedcontrol 113 and a handheld UID 114 in order to remotely control the arms104 and the surgical tools 107 (that are mounted on the distal ends ofthe arms 104.)

In some variations, the bedside operator 108 may also operate the system100 in an “over the bed” mode, in which the bedside operator 108 (user)is now at a side of the patient 106 and is simultaneously manipulating arobotically-driven tool (end effector as attached to the arm 104), e.g.,with a handheld UID 114 held in one hand, and a manual laparoscopictool. For example, the bedside operator's left hand may be manipulatingthe handheld UID to control a robotic component, while the bedsideoperator's right hand may be manipulating a manual laparoscopic tool.Thus, in these variations, the bedside operator 108 may perform bothrobotic-assisted minimally invasive surgery and manual laparoscopicsurgery on the patient 106.

During an example procedure (surgery), the patient 106 is prepped anddraped in a sterile fashion to achieve anesthesia. Initial access to thesurgical site may be performed manually while the arms of the roboticsystem 100 are in a stowed configuration or withdrawn configuration (tofacilitate access to the surgical site.) Once access is completed,initial positioning or preparation of the robotic system 100 includingits arms 104 may be performed. Next, the surgery proceeds with theremote operator 109 at the user console 102 utilizing the foot-operatedcontrols 113 and the UIDs 114 to manipulate the various end effectorsand perhaps an imaging system, to perform the surgery. Manual assistancemay also be provided at the procedure bed or table, by sterile-gownedbedside personnel, e.g., the bedside operator 108 who may perform taskssuch as retracting tissues, performing manual repositioning, and toolexchange upon one or more of the robotic arms 104. Non-sterile personnelmay also be present to assist the remote operator 109 at the userconsole 102. When the procedure or surgery is completed, the system 100and the user console 102 may be configured or set in a state tofacilitate post-operative procedures such as cleaning or sterilizationand healthcare record entry or printout via the user console 102.

In one embodiment, the remote operator 109 holds and moves the UID 114to provide an input command to move a robot arm actuator 117 in therobotic system 100. The UID 114 may be communicatively coupled to therest of the robotic system 100, e.g., via a console computer system 116.Representatively, in some embodiments, UID 114 may be a portablehandheld user input device or controller that is ungrounded with respectto another component of the surgical robotic system. For example, UID114 may be ungrounded while either tethered or untethered from the userconsole. The term “ungrounded” is intended to refer to implementationswhere, for example, both UIDs are neither mechanically nor kinematicallyconstrained with respect to the user console. For example, a user mayhold a UID 114 in a hand and move freely to any possible position andorientation within space only limited by, for example, a trackingmechanism of the user console 102. The UID 114 can generate spatialstate signals corresponding to movement of the UID 114, e.g. positionand orientation of the handheld housing of the UID, and the spatialstate signals may be input signals to control a motion of the robot armactuator 117. The robotic system 100 may use control signals derivedfrom the spatial state signals, to control proportional motion of theactuator 117. In one embodiment, a console processor of the consolecomputer system 116 receives the spatial state signals and generates thecorresponding control signals. Based on these control signals, whichcontrol how the actuator 117 is energized to move a segment or link ofthe arm 104, the movement of a corresponding surgical tool that isattached to the arm may mimic the movement of the UID 114. Similarly,interaction between the remote operator 109 and the UID 114 can generatefor example a grip control signal that causes a jaw of a grasper of thesurgical tool 107 to close and grip the tissue of patient 106.

The surgical robotic system 100 may include several UIDs 114, whererespective control signals are generated for each UID that control theactuators and the surgical tool (end effector) of a respective arm 104.For example, the remote operator 109 may move a first UID 114 to controlthe motion of an actuator 117 that is in a left robotic arm, where theactuator responds by moving linkages, gears, etc., in that arm 104.Similarly, movement of a second UID 114 by the remote operator 109controls the motion of another actuator 117, which in turn moves otherlinkages, gears, etc., of the robotic system 100. The robotic system 100may include a right arm 104 that is secured to the bed or table to theright side of the patient, and a left arm 104 that is at the left sideof the patient. An actuator 117 may include one or more motors that arecontrolled so that they drive the rotation of a joint of the arm 104, tofor example change, relative to the patient, an orientation of anendoscope or a grasper of the surgical tool 107 that is attached to thatarm. Motion of several actuators 117 in the same arm 104 can becontrolled by the spatial state signals generated from a particular UID114. The UIDs 114 can also control motion of respective surgical toolgraspers. For example, each UID 114 can generate a respective gripsignal to control motion of an actuator, e.g., a linear actuator, thatopens or closes jaws of the grasper at a distal end of surgical tool 107to grip tissue within patient 106.

In some aspects, the communication between the platform 105 and the userconsole 102 may be through a control tower 103, which may translate usercommands that are received from the user console 102 (and moreparticularly from the console computer system 116) into robotic controlcommands that are transmitted to the arms 104 on the robotic platform105. The control tower 103 may also transmit status and feedback fromthe platform 105 back to the user console 102. The communicationconnections between the robotic platform 105, the user console 102, andthe control tower 103 may be via wired and/or wireless links, using anysuitable ones of a variety of data communication protocols. Any wiredconnections may be optionally built into the floor and/or walls orceiling of the operating room. The robotic system 100 may provide videooutput to one or more displays, including displays within the operatingroom as well as remote displays that are accessible via the Internet orother networks. The video output or feed may also be encrypted to ensureprivacy and all or portions of the video output may be saved to a serveror electronic healthcare record system. It will be appreciated that theoperating room scene in FIG. 1 is illustrative and may not accuratelyrepresent certain medical practices.

In addition, in some aspects, surgical robotic system 100 may furtherinclude a tracking component 118 for tracking a characteristic of theuser, for example, the remote operator 109. The tracked characteristiccan, in turn, be used by the system 100 to automatically control anoperation of the surgical robotic system. For example, during ateleoperation mode of the surgical robotic system 100, in which the useris controlling the surgical tool 107 using the UID 114, the user shouldbe viewing the tool movement on display 115. In some cases, however, theuser may look away from the display 115 (intentionally orunintentionally) while still holding the UID 114 and controlling thesurgical tool 107. This introduces a risk since the user could move theUID 114 and, in turn, unintentionally move the tool 107 while notfocused on the display 115. Surgical robotic system 100 may thereforefurther include a tracking component 118 that tracks a characteristic ofthe user that can be used to determine whether the user is focused onthe display 115 and, in turn, intentionally operating the UID 114 andassociated tool 107, or not focused on the display such that ateleoperation mode should be disengaged or the system should transitionto a non-teleoperation mode. For example, in one aspect, the trackingcomponent 118 may be an eye tracker that can detect if the user islooking away from, or otherwise outside of, the display, based on thegaze of the user. If it is determined that the user gaze is outside ofthe display, and therefore the user may be looking away from thedisplay, the system may automatically pause the teleoperation mode sothat the user is temporarily prevented from using the UID 114 to movethe surgical tool 107. For example, teleoperation mode may be pauseduntil the tracking component 118 detects that the gaze is inside of thedisplay, and therefore the user is looking back at the display. Inaddition, the tracking component 118 may be used to detect additionalcharacteristics of the user, that if detected in addition to detectingthe user is looking away, will cause the surgical robotic system todisengage the teleoperation mode. For example, if it is determined theuser gaze is outside of the display and it is further determined thatthe user gaze is outside of the display for more than a maximum amountof time, the system may automatically disengage teleoperation mode. Inaddition, if it is determined the gaze is outside the display and theUID 114 is moving, the system may automatically disengage teleoperationmode.

It should be understood that “pausing” a teleoperation mode, is intendedto refer to a different system operation, with different consequences,than “disengaging” a teleoperation mode. In particular, pausing ateleoperation mode may be an operation that occurs when the systemdetermines the user may be briefly looking away from the display (e.g.,to reach another system component, readjust their position, etc), butthere is otherwise a high probability that the user is generally engagedand focused on the surgical operation or procedure. In this aspect, thesystem determines it is only necessary to temporarily prevent the UID114 from controlling the tool 107, and teleoperation mode is paused, butis not disengaged or transitioned entirely out of a teleoperation mode.In addition, since the user is otherwise engaged with the system andthere is little risk of an unintended action occurring, a relativelysmall input to the system may un-pause teleoperation mode. For example,if the input causing the system to pause teleoperation mode is adetermination or detection that the user gaze is outside of the displayfor not more than a maximum period of time, the input required toun-pause teleoperation mode may be a determination or detection that theuser gaze is once again inside the display.

“Disengaging” the teleoperation mode, on the other hand, is intended torefer to an operation which occurs when it is determined there is a highprobability that the user is not engaged or focused on the surgicaloperation or procedure, therefore it is inappropriate to continue inteleoperation mode. Disengaging teleoperation mode is therefore a morepermanent operation which prevents the UID 114 from controlling the tool107 until an intentional action or sequence of actions which clearlyindicate the user is now engaged and focused on the operation, anddesires to re-engage teleoperation mode, occurs. For example, aspreviously discussed, teleoperation mode may be disengaged when thesystem determines that the user gaze is outside of the display for morethan a maximum period of time, or outside of the display in combinationwith the detection of a movement of the UID. Such actions suggest a highprobability that the user is not focused on the procedure at hand, andit may not be appropriate to continue in teleoperation mode. The systemwill therefore automatically disengage the teleoperation mode so thatthe UID 114 cannot control the tool 107. To re-engage teleoperationmode, an intentional action or sequence of actions clearly indicatingthe user is focused on controlling the system is necessary. For example,the action(s) indicating the user is focused on the operation may bethat the user is facing the display, the user's chair is in a particularorientation, the UID is in a particular orientation, etc.

In addition, it is noted that the term “open display” is intended torefer to a display which is designed to allow the user to see outside ofthe display, for example with their peripheral vision, even whendirectly facing the display. In addition, in an open display, the usercan be a distance from the display screen, or not directly in front ofthe display, and still view a surgical procedure on the display.Therefore, in an open display system as disclosed herein, the fact thatthe user may not be close to the display or have their face directly infront of the display, would not necessarily be interpreted to mean theuser is distracted, or otherwise not paying sufficient attention tocontinue a surgical procedure. In the case of an open display, in whichthe user can turn their head and still see the display using theirperipheral vision, it is therefore important that turning of the headslightly not be considered a characteristic that will automaticallydisengage the teleoperation mode. Rather, an open architecture ordisplay surgical robotic system as disclosed herein will have sometolerance to such actions and allow the user to continue controlling asurgical tool. This is in contrast to a closed architecture system whichincludes, for example, a periscope with a completely immersive displaythat prevents the user from seeing outside of the display screen whenthey are facing the display and requires the user to be relatively closeto the display screen. For example, in the case of a periscope, the usermust have their face relatively close to, and facing, the display screento use it to view the surgical procedure in progress. If the user pullstheir head away from the display screen, or doesn't face the displayscreen, they can no longer see the display screen, therefore this wouldtypically be interpreted to mean the user is distracted or not payingsufficient attention to continue a surgical procedure.

Referring now to FIG. 2, FIG. 2 illustrates a side pictorial view of anexemplary display and tracking component. Representatively, display 115may include a tracking component 118 coupled to the display 115 in sucha manner that that it can track a characteristic of a user. For example,in one embodiment, tracking component 118 is an eye tracker that isoperable to track a gaze of a user relative to display 115. In thisaspect, tracking component 118 may be attached to, or integrated within,any portion of display 115 suitable for tracking a gaze of user 109. Forexample, tracking component 118 may be attached to a housing 202 ofdisplay 115, for example, a top wall, a bottom wall, or a side wall ofhousing 202, or integrated within a screen (not shown) mounted withinthe housing 202 of display 115. The tracking component 118 may includeone or more projector(s) and camera(s) which face the user and can beused to track a gaze of a user. Representatively, the projector(s) maycreate a pattern of near-infrared light on the eyes of the user, and thecamera(s) may take images of the user's eyes and the pattern. Thetracking component 118 may further be programmed to use this information(e.g., execute machine learning, image processing and/or mathematicalalgorithms) to determine where the user is looking based on a positionof each of the user's eyes and/or gaze point or location relative to oneanother, and display 115. For example, when the user 109 is positionedin front of display 115 as shown in FIG. 2, the tracking component 118may be configured to create a pattern of near-infrared light on the eyesof the user and take images of the user's eyes and the pattern when theuser's head is within a predetermine sensing range 204 of tracker 118.This information may, in turn, be used by a surgical robotic systemprocessing component to determine whether the gaze 206 of the user 109is at a gaze point 208 inside or outside of display 115, the user islooking toward or away from display 115, and whether to pause ordisengage a teleoperation mode.

Representatively, in one aspect, the surgical robotic system implementsa two-fold process for determining whether teleoperation mode should bepaused or disengaged, based on the user's gaze. For example, the processmay generally include a first sequence of operations used to determineif the user is looking toward or away from the display, also referred toin some embodiments as a gaze-off-screen operation. The process mayfurther include a second sequence of operations used to determine whenteleoperation mode should be paused and when teleoperation should bedisengaged, based in part, on the outcome of the gaze-off-screenoperation and the probability that the user is looking on/off display.The second sequence of operations may be referred to, in someembodiments, as a gaze-teleop-interlock operation because it may includeone or more interlocks (e.g. conditions to determine if the user isfocused on teleoperation) that may be considered when pausing ordisengaging teleoperation mode.

Referring now in more detail to the gaze-off-screen operation, thisoperation may be used to determine a probability or likelihood that theuser is looking at the display or is looking away from the display, byevaluating the gaze data collected by the eye tracker. It is furthernoted that the gaze-off-screen operation may be configured to discardblinks, and work for different users with, for example, different eyeanatomy, no glasses, glasses, and/or contacts. The data collected by theeye tracker during this operation may be considered “valid”, meaning theeye(s) are detected by the eye tracker, or “invalid”, meaning the eye(s)are not detected by the eye tracker. For example, the eye tracker isprogrammed to detect the user gaze and/or gaze point within apredetermined sensing or gaze range, which may, for example, bedetermined based on a size of the display (with some tolerance rangeoutside of the display). When the eye tracker cannot recognize or detectthe gaze or pupil of one or both of the user's eyes within thispredetermined sensing or gaze range, it generates an invalid signal. Forexample, when the user's head is turned such that one or both eyes ofthe user are at an angle to the display or the user is blinking, the eyetracker may not be able to recognized or detect the pupil or gaze of theuser, and generate an invalid signal. In addition, the eye tracker maygenerate an invalid signal if both eyes are detected, but the gaze pointbetween the eyes is unusually large or otherwise inconsistent with atypical user gaze. A valid signal, on the other hand, may be generatedby the eye tracker when the eye tracker can detect one or both eyes,and/or the gaze point between both eyes is normal or otherwiseconsistent with a typical user gaze.

A probability or likelihood that the user gaze is on screen or offscreen is then determined based on the received valid and/or invalidsignals. For example, if the eye tracker determines that both eyes areinvalid, or both eyes are valid, but the distance between calculatedscreen points of the corresponding gaze points is too large, thissuggests a blink, the tracker is unable to see the eyes orinconsistencies for both eyes. In such cases, the probability the gazeis off-screen increases and the probability the gaze is on-screendecreases. In addition, if at least one eye is valid, and the gaze isinside the display, the probability the gaze is on-screen increases andthe probability the gaze is off-screen decreases. Moreover, if two eyesare valid, and the gaze for both eyes are inside the display, thegaze-off-screen operation determines the gaze is on-screen and theprobability the gaze is on-screen increases even more. On the otherhand, if both eyes are valid and outside the display, or one is validand outside the display, this suggests both eyes are looking outside thedisplay or at least one eye is looking outside the display, andtherefore the probability the gaze is off-screen is high.

In addition, to avoid false positives due to, for example, gaze lossfrom blinking, squinting or inaccurate gaze data, the gaze-off-screenoperation may further consider the amount of time an invalid signal isdetected, the gaze point or gaze path detected prior to the invalidsignal, a location and/or orientation of the head and/or face, and/ortrack eye ball locations. Based on all the data collected, theprobability or likelihood the user is looking off-screen versus theprobability the user is looking on-screen is weighed to evaluate therisks of continuing teleoperation mode and need for further action(e.g., pause or disengage teleoperation mode).

In particular, the gaze-teleop-interlock operation, based at least inpart on the outcome of the gaze-off-screen operation, may determinewhether to pause or disengage teleoperation. For example, the gazeon-screen and gaze-off screen probabilities may be weighed against oneanother as the data is received and used to evaluate whether the user islooking toward the display or away from the display, and in turn,whether teleoperation should be paused (e.g., probability user islooking toward the display>probability user is looking away from thedisplay) or disengaged (e.g., probability user is looking away from thedisplay>probability user is looking toward the display) by thegaze-teleop-interlock operation. Moreover, in an open architecturesystem as contemplated herein, interactions between the user and staffare expected and the system should support this behavior to some extent.In order to do that, when the gaze-off-screen operation determines theuser is looking off screen, the gaze-teleop-interlock operationdetermines whether the user is looking away for more than a blink or aquick glance, and if so, causes the system to pause the teleoperationmode. When it is determined that he user is looking back at the display(e.g., a valid signal is detected), teleoperation is un-paused, and theuser can immediately control the tools. On the other hand, if the userlooks away for a significant amount of time, or the user moves the UID asignificant amount while looking away, the gaze-teleop-interlockoperation determines it is not appropriate for the user to continueoperating the tool and causes the system to disengage teleoperation.

Exemplary processes for determining if the user gaze is inside oroutside of the display 115, the user is looking toward or away from thedisplay and/or whether to pause or disengage a teleoperation mode, willnow be discussed in more detail in reference to FIG. 3-FIG. 6.

Representatively, referring now to FIG. 3, FIG. 3 illustrates a blockdiagram of an exemplary process for disengaging a teleoperation modebased on a user gaze. In particular, process 300 may include an initialoperation of receiving a gaze input (block 302). The gaze input may bethe user gaze, gaze point and/or gaze location information detected bythe tracking component 118, more specifically an eye tracking component,as previously discussed. This information is then used to determinewhether the user gaze is outside or inside of the display (block 304).The process for determining whether the user gaze is outside or insidethe display based on this information will be discussed in more detailin reference to FIG. 4.

Returning now to FIG. 3, if it is determined that the user gaze isinside the display, teleoperation mode continues (block 306). If,however, it is determined at operation 304 that the user gaze is outsideof the display, teleoperation mode is paused (block 308). As previouslydiscussed, pausing of the teleoperation mode temporarily prevents theUID 114 from controlling the tool 107 until, for example a reverseoperation is detected. In this aspect, tracking component 118 maycontinue to check for a user gaze, for example, every 10-20milliseconds, to determine whether the user gaze is outside of thedisplay for more than a maximum amount of time (block 310). If it is notoutside the display for more than the maximum amount of time, in otherwords a user gaze inside of the display is detected before the maximumamount of time has passed, process 300 returns to operation 306 andteleoperation mode will be automatically un-paused so that the user cancontinue teleoperation mode and control the instrument using the UID.The maximum amount of time may be, for example, the amount of time ittakes a person to blink. For example, the average blink may last from100 milliseconds to 300 milliseconds, therefore the maximum amount oftime may be 100 milliseconds, 200 milliseconds, 300 milliseconds. Instill further embodiments, the maximum amount of time may correspond toan amount of time that is determined to be inappropriate to continuewith operation of the system. For example, the maximum amount of timemay be about 5 seconds or more.

If, on the other hand, the user gaze is determined to be outside of thedisplay for more than the maximum amount of time (e.g., more than 100milliseconds, more than 200 milliseconds, more than 300 milliseconds ormore than 5 seconds), teleoperation mode may be disengaged (block 314).In addition, in some cases, process 300 may further detect a UID motioninput and use the motion input, in combination with the gaze input, todetermine whether to disengage teleoperation mode. Representatively,process 300 may include the further operation of detecting a motion ofthe UID using a UID motion sensor associated with the system anddetermining whether the motion input is greater than a maximumtranslation (block 312). If this UID motion input is not greater than amaximum translation, teleoperation mode may remain paused. If the UIDmotion input is greater than a maximum translation, teleoperation modeis disengaged (block 314). The “maximum translation” may refer to, forexample, a maximum acceptable range of motion of the UID which can occurwhen the user is not looking at a display. For example, a relativelysmall movement of the UID such as would occur when the user is adjustingtheir hand and/or wrist position while holding the UID would beconsidered an acceptable range. Alternatively, a relatively largemovement of the UID such as would occur when the user engages in anunintentional movement of the UID or has dropped the UID would begreater than the maximum acceptable range of motion of the UID and causethe system to disengage teleoperation. Once disengaged fromteleoperation mode, process 300 determines whether an active engagementinput is received (block 316). If an active engagement input isreceived, process 300 will return to operation 306 and teleoperationmode may be re-engaged so that teleoperation mode can continue. If anactive engagement input is not received, process 300 will remaindisengaged from teleoperation mode until the active engagement input isreceived. As previously discussed, an active engagement input may be anintentional action or sequence of actions that indicate a desire by theuser to re-engage teleoperation mode (e.g., control instrument 107 usingUID 114). For example, the action(s) indicating the user is focused onthe operation may be that the user is facing the display, the user'schair is in a particular orientation, the UID is in a particularorientation, etc.

FIG. 4 is a block diagram of an exemplary operation for determiningwhether a user gaze is outside a display, in accordance with anembodiment. Process 400 may include an initial operation of receiving agaze input (block 402), which is similar to operation block 302 ofprocess 300. The remaining operations of process 400 are then used todetermine whether the gaze of a user is outside or inside a display(e.g., operation block 304 of process 300). Representatively, once thegaze input is received (block 402), it is determined whether the gazeinput is valid or invalid (block 404), as previously discussed.

If a valid gaze input is received, process 400 continues to operation406 to determine whether both eyes are valid. If both eyes are valid,process 400 continues to operation 410 where a distance between a gazepoint for both eyes is analyzed to determine whether it is greater thana maximum distance. The maximum distance may be a distance between agaze point of each eye of the user which is considered within the rangeof a normal user gaze looking at the display. An exemplary maximumdistance between a gaze point of each eye is illustrated in FIG. 5.Representatively, FIG. 5 illustrates a display 115 including a screen502 and a frame 504 around the screen 502. A gaze point or location of aleft eye is illustrated by point 508 and a gaze point or location of aright eye is illustrated by point 510. A measurement of the distancebetween gaze points 508, 510 is illustrated by line 512. If thisdistance 512 between a gaze point for both eyes is not greater than themaximum distance, the user gaze is considered consistent with a userlooking at the display, and it is determined that the user gaze isinside the display (block 412). On the other hand, if the distance 512between the gaze point for both eyes is greater than the maximumdistance, the gaze is not consistent with a user looking at the display,and it is determined that the user gaze is outside the display (block414).

If both eyes are not invalid (e.g., one eye is invalid and one eye isvalid), operation 400 continues to operation block 416 where it isdetermined whether a gaze point or location of at least one valid eye isoutside of the size of the display. For example, as previously discussedin reference to FIG. 5, display 115 may include screen 502 and frame 504around the screen 502, such that the size of the display corresponds toan area or region at least as big as the frame surrounding the screen.In some cases, the size of the display may extend a distance outside ofthe frame which accounts for the peripheral vision of the user. The sizeof the display including this tolerance range outside of the frame isillustrated by dashed line 506. In other words, the gaze point orlocation could be slightly outside of the frame (e.g., within range 506)but a peripheral vision of the user would allow the user to still seethe screen, therefore the user gaze point or gaze location, and in turnthe user gaze, will still be considered inside the size of the display(or inside the display). Alternatively, if the gaze point or gazelocation is outside of the range 506, the gaze point or gaze location,and in turn the user gaze, is considered outside of the size of thedisplay (or outside of the display). If the gaze location of the atleast one valid eye is not determined to be outside of the size ofdisplay 506, process 400 continues to operation block 412 and determinesthe user gaze is inside of the display. If, however, the gaze point orlocation of one valid eye is outside of the size of display, processdetermines the user gaze is outside of the display at operation block414.

If, on the other hand, an invalid gaze input is received at operationblock 404, process 400 continues to operation block 408 to determinewhether both eyes are invalid. If both eyes are not invalid (e.g., oneeye is invalid and one eye is valid), operation 400 continues tooperation block 416 where it is determined whether a user gaze, gazepoint or location of at least one valid eye is outside of the size ofthe display as previously discussed.

If, on the other hand, it is determined at operation block 408 that botheyes are invalid, process 400 continues to operation block 418 where itis determined whether both eyes are invalid for more than a maximumamount of time. The maximum amount of time may be, for example, theamount of time that an average blink lasts. For example, the maximumamount of time may be 100 milliseconds, 200 milliseconds or 300milliseconds. In some cases, the maximum amount of time may be around 5seconds. If both eyes are not invalid for more than the maximum amountof time (e.g., a valid input is received in 100 milliseconds, 200milliseconds or 300 milliseconds, or less), process determines the userwas blinking (block 422). Blinking is treated as though the user isstill looking at the screen, therefore process 400 returns to operationblock 402 and continues the gaze input analysis.

If, on the other hand, both eyes are invalid for more than the maximumamount of time (e.g., more than 100 milliseconds, more than 200milliseconds, or more than 300 milliseconds), the possibility that theinvalid signal is due to, for example, the user blinking, squinting,inaccurate data, etc. is unlikely. Instead, it is more likely the casethat the user is looking away from the display. To confirm blinking canbe ruled out, process 400 may further evaluate the last gaze location orgaze path, head location/orientation and/or eyeball location of the userat operation block 420.

Representatively, FIG. 6 illustrates one exemplary operation forevaluating the last gaze point or location, or gaze path of the user.The last gaze point or location is illustrated in FIG. 6 as point 510and the gaze path is illustrated as dashed line 602. If the gaze point510 and gaze path 602 are detected by the eye tracker as moving towardthe border of the display 506 prior to receiving the invalid signal,this suggests the user is turning their head away from the display andcan be used to rule out blinking, squinting and/or inaccurate data.Alternatively, if the gaze point 510 is fixed prior to receiving theinvalid signal, this suggests the user is not turning their head awayfrom the display and there is still a possibility that the invalidsignal is due to the user blinking, squinting and/or inaccurate data.

In some cases, for example where it is not entirely clear from the gazelocation/path information whether the invalid input is due to the userblinking or looking away from the display, a head location/orientationinput may further be received and considered in operation 420. The headlocation input may be received, for example, from a tracking componentthat tracks a head location and/or face of the user. For example,tracking component 118 may be, or may further include, a head tracker,with similar tracking components (e.g., projector(s) and camera(s))which can be used to detect when the user's nose is pointing toward thedisplay. The tracker can further track the user's face and indicate whenthe ace is no longer visible. For example, if a head location inputindicating the user's face is no longer visible or not aligned with thedisplay (e.g., the user's nose is not facing the display) is received,process 400 confirms blinking may be ruled out. Still further, in somecases, the tracking component can track the eyeball location instead of,or in addition to, head tracking, and be used to distinguish betweenlooking away from the display and blinking. For example, the trackingcomponent can track whether the last detected eyeball location wasfixed, or was moving toward the border of the display 506, todistinguish between blinking and looking away from the display.

Operation 420, however, is optional therefore in some cases, operation420 is omitted and process 400 proceeds from operation block 418 tooperation block 414 where it is concluded that the gaze is outside ofthe display.

Referring now to FIG. 7, FIG. 7 is a block diagram of a computer portionof a surgical robotic system, which is operable to implement thepreviously discussed operations, in accordance with an embodiment. Theexemplary surgical robotic system 700 may include a user console 102, acontrol tower 103, and a surgical robot 120. The surgical robotic system700 may include other or additional hardware components; thus, thediagram is provided by way of example and not a limitation to the systemarchitecture.

As described above, the user console 102 may include console computers711, one or more UIDs 712, console actuators 713, displays 714, a UIDtracker 715, foot pedals 716, and a network interface 728. A user orsurgeon sitting at the user console 102 can adjust ergonomic settings ofthe user console 102 manually, or the settings can be automaticallyadjusted according to user profile or preference. The manual andautomatic adjustments may be achieved through driving the consoleactuators 713 based on user input or stored configurations by theconsole computers 711. The user may perform robot-assisted surgeries bycontrolling the surgical robot 120 using one or more master UIDs 712 andfoot pedals 716. Positions and orientations of the UIDs 712 arecontinuously tracked by the UID tracker 715, and status changes arerecorded by the console computers 711 as user input and dispatched tothe control tower 103 via the network interface 718. Real-time surgicalvideo of patient anatomy, instrumentation, and relevant software appscan be presented to the user on the high resolution 3D displays 714including open or immersive displays.

The user console 102 may be communicatively coupled to the control tower103. The user console also provides additional features for improvedergonomics. For example, the user console may be an open architecturesystem including an open display, although an immersive display, in somecases, may be provided. Furthermore, a highly-adjustable seat forsurgeons and master UIDs tracked through electromagnetic or opticaltrackers are included at the user console 102 for improved ergonomics.

In addition, as previously discussed, user console may include atracking component for tracking a characteristic of a user, which can inturn be used to prevent accidental tool motion, for example, by pausingor disengaging teleoperation when the user's gaze is not engaged in thesurgical site on the open display for over a predetermined period oftime. Representatively, user console 102 may include a gaze tracker 710that tracks a gaze of the user to determine whether to pause ordisengage the teleoperation mode, as previously discussed in referenceto FIG. 3 and FIG. 4. In addition, gaze tracker 710 may include a headtracking, eyeball tracking and/or surgical component tracking mechanismthat can further be used to determine whether to pause or disengage theteleoperation mode as previously discussed.

The control tower 103 can be a mobile point-of-care cart housingtouchscreen displays, computers that control the surgeon'srobotically-assisted manipulation of instruments, safety systems,graphical user interface (GUI), light source, and video and graphicscomputers. As shown in FIG. 7, the control tower 103 may include centralcomputers 731 including at least a visualization computer, a controlcomputer, and an auxiliary computer, various displays 733 including ateam display and a nurse display, and a network interface 728 couplingthe control tower 103 to both the user console 102 and the surgicalrobot 120. The control tower 103 may offer additional features for userconvenience, such as the nurse display touchscreen, soft power andE-hold buttons, user-facing USB for video and still images, andelectronic caster control interface.

The auxiliary computer may also run a real-time Linux, providinglogging/monitoring and interacting with cloud-based web services.

The surgical robot 120 may include an articulated operating table 724with a plurality of integrated arms 722 that can be positioned over thetarget patient anatomy. A suite of compatible tools 723 can be attachedto or detached from the distal ends of the arms 722, enabling thesurgeon to perform various surgical procedures. The surgical robot 120may also comprise control interface 725 for manual control of the arms722, table 724, and tools 723. The control interface can include itemssuch as, but not limited to, remote controls, buttons, panels, andtouchscreens. Other accessories such as trocars (sleeves, sealcartridge, and obturators) and drapes may also be needed to performprocedures with the system. In some variations, the plurality of thearms 722 includes four arms mounted on both sides of the operating table724, with two arms on each side. For certain surgical procedures, an armmounted on one side of the table can be positioned on the other side ofthe table by stretching out and crossing over under the table and armsmounted on the other side, resulting in a total of three arms positionedon the same side of the table 724. The surgical tool can also comprisetable computers 721 and a network interface 728, which can place thesurgical robot 120 in communication with the control tower 103.

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope of the invention as set forth in thefollowing claims. The specification and drawings are, accordingly, to beregarded in an illustrative sense rather than a restrictive sense.

What is claimed is:
 1. A method for disengaging a surgical instrument ofa surgical robotic system, the method comprising: receiving a gaze inputfrom an eye tracker that tracks a gaze of a user relative to a displayassociated with the surgical robotic system; determining, by one or moreprocessors communicatively coupled to the eye tracker, whether the gazeinput indicates the gaze of the user is outside or inside of thedisplay; in response to determining the gaze input indicates the gaze ofthe user is outside of the display, determining an amount of time thegaze of the user is outside of the display; in response to determiningthe gaze of the user is outside of the display for less than a maximumamount of time, pause the surgical robotic system from a teleoperationmode such that a user interface device of the surgical robotic system isprevented from controlling the surgical instrument until a gaze inputindicating the gaze of the user is inside of the display is received;and in response to determining the gaze of the user is outside of thedisplay for more than the maximum amount of time, disengage the surgicalrobotic system from the teleoperation mode such that the user interfacedevice is prevented from controlling the surgical instrument until anactive engagement input is received.
 2. The method of claim 1 whereindetermining the gaze input indicates the gaze of the user is outside ofthe display comprises: receiving a valid gaze input indicating that agaze for both eyes of the user are detected by the eye tracker; anddetermining a distance between the gaze for both eyes is outside of amaximum distance associated with a size of the display.
 3. The method ofclaim 1 wherein determining the gaze input indicates the gaze of theuser is outside of the display comprises: receiving an invalid gazeinput indicating a gaze of at least one eye of the user is undetectableby the eye tracker; and determining a gaze path or a gaze location ofthe at least one eye is moving toward a border of the display.
 4. Themethod of claim 1 wherein determining the gaze input indicates the gazeof the user is outside of the display comprises: receiving a headlocation input from a head tracker associated with the surgical roboticsystem; and determining, based on the head location input, a nose of theuser is not facing the display or a face of the user is undetectable. 5.The method of claim 1 wherein determining the gaze of the user is insidethe display comprises: receiving a valid gaze input indicating that agaze of at least one eye of the user is detected by the eye tracker; anddetermining a location of the gaze is within the display.
 6. The methodof claim 1 further comprising: receiving a movement input from the userinterface device of the surgical robotic system; and in response toreceiving the movement input when the surgical robotic system is paused,disengage the surgical robotic system.
 7. The method of claim 1 whereinthe active engagement input indicates a user is facing the display, asurgical chair is in a particular orientation, or the user interfacedevices is in a particular orientation.
 8. The method of claim 1 whereinthe maximum amount of time is more than 100 milliseconds.
 9. A surgicalrobotic system, comprising: a surgical instrument; a user consolecomprising a display, an eye tracker for tracking a gaze of a user withrespect to the display, and a user interface device; and one or moreprocessors communicatively coupled to the surgical instrument and theuser console, the processors configured to: receive a gaze input fromthe eye tracker; determine whether the gaze of the user is outside orinside of the display based on the gaze input; and pause and/ordisengage the surgical robotic system from a teleoperation mode when theuser gaze is outside the display such that the user interface device isprevented from controlling the surgical instrument.
 10. The surgicalrobotic system of claim 9 wherein the display is an open displaycomprising a screen and a frame surrounding the screen.
 11. The surgicalrobotic system of claim 9 wherein the user interface device is aportable handheld user input device that is mechanically ungrounded withrespect to the user console.
 12. The surgical robotic system of claim 9further comprising a head tracker, and wherein determining whether thegaze of the user is outside or inside the display is based on a headlocation input from the head tracker.
 13. The surgical robotic system ofclaim 9 wherein the gaze of the user is determined to be outside of thedisplay when a gaze for both eyes of the user are detected by the eyetracker; and a distance between the gaze for both eyes is outside of amaximum distance associated with a size of the display.
 14. The surgicalrobotic system of claim 9 wherein the gaze of the user is determined tobe outside of the display when a gaze of at least one eye of the user isundetectable by the eye tracker; and a gaze path or a gaze location ofthe at least one eye is moving toward a border of the display.
 15. Thesurgical robotic system of claim 9 wherein the gaze of the user isdetermined to be outside of the display when a head location input froma head tracker associated with the surgical robotic system indicates anose of the user is not facing the display or a face of the user isundetectable.
 16. The surgical robotic system of claim 9 wherein thegaze of the user is determined to be inside the display when a gaze ofat least one eye of the user is detected by the eye tracker; and alocation of the gaze is within the display.
 17. The surgical roboticsystem of claim 9 further comprising a user interface device motionsensor, and wherein the surgical robotic system is paused and/ordisengaged from a teleoperation mode when a motion of the user interfacedevice greater than a maximum translation is detected.
 18. The surgicalrobotic system of claim 9 wherein in response to determining the gaze ofthe user is outside of the display for less than a maximum amount oftime, the surgical robotic system is paused from a teleoperation modesuch that a user interface device of the surgical robotic system isprevented from controlling the surgical instrument until a gaze inputindicating the gaze of the user is inside of the display is received.19. The surgical robotic system of claim 18 wherein the maximum amountof time is more than 100 milliseconds.
 20. The surgical robotic systemof claim 9 wherein in response to determining the gaze of the user isoutside of the display for more than a maximum amount of time, thesurgical robotic system is disengaged from the teleoperation mode suchthat the user interface device is prevented from controlling thesurgical instrument until an active engagement input is received.